Data Collecting Method, Data Collecting Apparatus and Network Management Device

ABSTRACT

The present invention provides a data collection method and apparatus and a network management device. The method includes: a network management device collecting data files to be processed reported by a network element device; assigning the data files to be processed as a plurality of tasks; adding the assigned tasks into a task queue and extracting tasks from the task queue one by one for processing. According to the present invention, the task work load can be automatically adjusted according to the computer configuration and parameter configuration, and the maximum efficiency of data processing can be achieved under different scenarios.

TECHNICAL FIELD

The present invention relates to a configurable and schedulable datacollection method and apparatus and a network management device.

BACKGROUND OF THE RELATED ART

The concepts and drawbacks of two existing data collection methods areas follows.

1. All the data files are reported to a certain directory designated bymanager by the person being managed, and all the data files under thisdirectory are collected and parsed during the data collection.

The collection and parse procedures for different products aredifferent, which can be generally divided into two types:

1) single thread running: the processing efficiency thereof is low, onceabnormality occurs to the thread, it cannot ensure all the data to beprocessed;

2) multi-thread running: resource competition of multiple threads mayappear, the overhead of the synchronization method will be increasedduring this competition, and the efficiency is reduced.

2. The data files generated by different persons being managed arereported to the designated directory for this person being managed inthe manager, and the data files under all the directories are collectedand parsed.

The directory can be divided into multiple levels, and there can be aplurality of parallel subdirectories in each directory level. All thedata are placed on the leaf directory (i.e. directory for person beingmanaged), each directory for person being managed is stored in differentupper level directory according to a certain rule; and so on, the upperlevel directory can also be stored in an further upper level directoryaccording to a certain rule.

The collection and parse procedures of different products are different,which can also be generally divided into two types:

1) single thread running: the processing efficiency is also low, and itwill cause the execution of magnetic disk IO operation to occupy a largeamount of time and resources for processing when the single threadaccess directory layer is deep;

2) multi-thread running: each thread being responsible for theprocessing under a non-leaf directory may cause the working loads ofeach thread to be uneven; and multiple threads being responsible for anon-leaf directory may further cause thread resource waste underdirectories with less data in addition to forming competition.

In addition, the above two existing concepts also have the followingdrawbacks:

the directory for data storage becomes the basis for processing threadcreation, startup and stopping, which will cause problems that theworking loads of the processing threads are uneven and monitoring cannotbe carried out uniformly.

SUMMARY OF THE INVENTION

The present invention provides a data collection method and apparatusand a network management device, in which the task work load can beautomatically adjusted according to the computer configuration andparameter configuration to achieve the maximum efficiency of dataprocessing under different scenarios.

In order to solve the above technical problem, the present inventionprovides a data collection method, comprising:

a network management device collecting data files to be processedreported by a network element device;

assigning the data files to be processed as a plurality of tasks; and

adding the plurality of tasks into a task queue and extracting tasksfrom the task queue one by one for processing.

Preferably, the above method has the following features: the step ofassigning the data files to be processed as a plurality of taskscomprises:

evaluating quantity or capacity of the data files to be processed;

evaluating local calculation ability;

obtaining local configuration parameters;

calculating task payload according to the quantity or capacity of thedata files to be processed, the local calculation ability and theconfiguration parameters; and

assigning the data files to be processed as a plurality of tasks basedon the task payload.

Preferably, the above method has the following features: the localcalculation ability comprises: local CPU processing speed and memorycapacity.

Preferably, the above method has the following features: theconfiguration parameters comprises: the number of threads and themaximum activation time of the threads.

The present invention also provides a network management device,comprising:

a data collection apparatus, which is configured to: obtain data filesto be processed reported by a network element device; assign the datafiles to be processed into a plurality of tasks; add the plurality oftasks into a task queue, and extract tasks from the task queue one byone for processing.

Preferably, the above network management device has the followingfeatures:

the data collection apparatus is configured to assign the data files tobe processed into a plurality of tasks by way of the following manner:evaluating quantity or capacity of the data files to be processed;evaluating local calculation ability; obtaining local configurationparameters; calculating task payload according to the quantity orcapacity of the data files to be processed, the local calculationability and the configuration parameters; and assigning the data filesto be processed as a plurality of tasks based on the task payload.Preferably, the above network management device has the followingfeatures: the local calculation ability comprises: local CPU processingspeed and memory capacity.

Preferably, the above network management device has the followingfeatures: the configuration parameter comprises: the number of threadsand the maximum activation time of the threads.

The present invention also provides a data collection apparatus,comprising:

a task scheduling module being configured to obtain data files to beprocessed reported by a network element device, assign the data files tobe processed into a plurality of tasks, and send the plurality of tasksto a thread pool module; and

the thread pool module being configured to add the plurality of tasksreceived into a task queue, and extract tasks from the task queue one byone for processing.

Preferably, the above data collection apparatus has the followingfeatures: the task scheduling module comprises:

a data evaluation unit being configured to collect data files to beprocessed reported by a network element device, evaluate quantity orcapacity of the data files to be processed, and send the evaluatedquantity or capacity to a calculation unit;

a calculation ability evaluation unit being configured to evaluate localcalculation ability and send the evaluated local calculation abilityinformation to the calculation unit;

a configuration parameter unit being configured to obtain configurationparameters and send the obtained configuration parameters to thecalculation unit;

the calculation unit being configured to calculate task payloadaccording to the quantity or capacity of the data files to be processed,the local calculation ability and the configuration parameters, and sendthe calculated task payload to a task assignment unit; and

the task assignment unit being configured to, after receiving the taskpayload, assign the data files to be processed as a plurality of tasksbased on the task payload, and send the plurality of tasks to the threadpool module.

Preferably, the above data collection apparatus has the followingfeatures: the local calculation ability comprises: local CPU processingspeed and memory capacity.

Preferably, the above data collection apparatus further comprises aconfiguration item module being configured to define configurationparameters, wherein the configuration parameters comprises: the numberof threads and the maximum activation time of the threads; theconfiguration parameter unit being configured to obtain theconfiguration parameters from the configuration item module.

The data collection method and apparatus and network management deviceprovided by the present invention have the following beneficial effects:

1. the task work load can be automatically adjusted according to thecomputer configuration and parameter configuration, and the maximumefficiency of data processing can be achieved under different scenarios;

2. when the task scheduling module divides tasks, it will not cause theworking ranges of different tasks to be overlapped, eliminating thepossibility of resource competition, and improving the task processingefficiency;

3. the content of a task is merely limited to the access to one leveldirectory, improving the IO operation efficiency of entering or exitingthe directory for the magnetic disk;

4. the configuration file manages the scale of the thread pool, whichcan improve the efficiency of data collection function without losingsystem efficiency;

5. the configuration file manages the activation time of the thread,reducing the possibility of abnormality occurring during the long timerunning of the thread;

6. although some processing time will be lost due to a large amount offile operations during task scheduling, the loss brought thereby is farless than the time saved by the subsequent processing;

7. the task scheduling module can assign tasks rationally according tothe computer configuration in real time, avoiding the situations thatthe working load of the computer is over large or the computer is notfully utilized.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of a network management device accordingto the embodiments of the present invention.

FIG. 2 is a schematic diagram of a data collection apparatus accordingto the embodiments of the present invention.

FIG. 3 is a schematic diagram of a task scheduling module according tothe embodiments of the present invention.

FIG. 4 is a flowchart of a data collection method according to theembodiments of the present invention.

PREFERRED EMBODIMENTS OF THE PRESENT INVENTION

Hereinafter, the technical solution provided by the embodiments of thepresent invention will be further described in conjunction with theaccompanying drawings. It shall be understood that the preferredembodiments here described are merely for explaining the presentinvention but not as a limit.

FIG. 1 is a schematic diagram of a network management device accordingto the embodiments of the present invention, and as shown in FIG. 1, thenetwork management device according to this embodiment includes a datacollection apparatus, which has the following functions:

obtain data files to be processed (which may refer to parsing the datafile, and may also refer to forwarding the data files to the upper levelof the network administrator) reported by a network element device;evaluate the quantity or capacity of the data files to be processed,calculate task payload according to the local calculation ability andconfiguration parameter, and then assign the data files to be processedinto a plurality of tasks according to the task payload. Variousassigned tasks are added into a task queue, then they are extracted fromthe task queue one by one for execution, and the content of eachextracted task is a part of data files to be processed.

FIG. 2 is a schematic diagram of a data collection apparatus accordingto the embodiments of the present invention, and as shown in FIG. 2, thedata collection apparatus in this embodiment includes: a trigger, a taskscheduling module, a thread pool module, a subsequent processing moduleand a configuration item module, wherein the main function of eachmodule is as follows.

The trigger is mainly used for triggering the task scheduling module tostart working.

The task scheduling module is used for collecting data files to beprocessed, such as collecting all the data files to be processed underall the leaf directories under a non-leaf directory, dynamicallyadjusting the working load of each thread according to thequantity/capacity of the data file, the computer configuration (such aslocal calculation ability) and the configuration item parametersextracted from the configuration item module, assigning a task, andsending the assigned task to the thread pool module.

The task scheduling module can automatically adjust the task work loadaccording to the computer configuration and the parameter configurationand achieve the maximum efficiency of the data processing underdifferent scenarios; when the task scheduling module assigns tasks, itwill not cause the working ranges of different tasks to be overlapped,which eliminates the possibility of resource competition, and improvesthe task processing efficiency; the assigned tasks are: the content of atask is merely limited to the access to one level directory (i.e. leafdirectory, referring to a plurality of directories on the same level andbelonging to the same parent directory), which improves the Io operationefficiency of entering or exiting the directory for the magnetic disk.

The thread pool module consists of a plurality of activated threadsprocessing the tasks and a task queue, the thread pool module adds thereceived task into the task queue, and the activated thread obtains atask from the task queue for processing. The activated thread firstlyextracts a task from the head of the task queue, obtains the informationabout the data file to be processed from the task, and then processesthe data file according to the corresponding information. The processingcan include: parsing the data file, forwarding the data file, but notlimited to these two processing modes. If the data file processingresult during task execution is not the end of the data flow or it needsfurther processing, then the task execution result has to be sent to thesubsequent processing module for further processing.

The configuration item module provides the definition of the datacollection function related parameter by way of an XML (ExtensibleMarkup Language) file or in the code. The configuration item includes:the number of threads of the thread pool module, the maximum activationtime of the threads, scheduling according to the quantity or capacity ofthe file, etc.

The configuration item module manages the scale of the thread pool,which can improve the efficiency of data collection function withoutlosing system efficiency; it provides a “the number of threads”parameter by way of the configuration item, and selects how manyactivated processes can be generate, i.e. the scale of the thread pool,according to this parameter during the initialization of the threadpool.

The configuration item module manages the activation time of the thread,reducing the possibility of abnormality occurring during the long timerunning of the thread. In particular, the “the activation time of thethread” parameter can be provided by way of the configuration item. Thisparameter decides the maximum running time of a thread. For example,this parameter is set as 1 minute, then the maximum time for a thread toexecute a task is 1 minute, and the thread will automatically stop theexecution of the task when it exceeds 1 minute and adds the data filethe processing of which is not completed in the task into the taskqueue. In order to avoid increasing possibilities of thread abnormalitydue to the continuous long time running of the thread, this parameter isset to reduce the possibility of thread abnormality.

The subsequent processing module provides the subsequent processingoperations, such as operation of writing into a database and forwardingoperation. It can also be designed as thread pool plus data queueaccording to the processing capacity, i.e. employing the operation modein the manner of one thread pool and a data queue (herein, the threadpool and data queue do not refer to the above thread pool and data queuebut refer to a general concept); however, if the processing capacity issmall, then it can be designed as a single thread plus data queue. Itshall be noted not to design it with the thread in the thread pool assynchronization manner. The subsequent processing operation may includewriting the data into a database or data forwarding operation, etc. butnot limited to these two types.

The structure of the task scheduling module in the data collectionapparatus in this embodiment is as shown in FIG. 3, including: a dataevaluation unit, a calculation unit, a calculation ability evaluationunit, a configuration parameter unit and a task assignment unit.

The data evaluation unit is configured to obtain data files to beprocessed, such as collecting the data files under all the leafdirectories under a non-leaf directory, evaluating the data files to beprocessed, such as evaluating the quantity or capacity of the datafiles, and send the evaluated quantity and/or capacity of the data filesto the calculation unit.

The calculation ability evaluation unit is configured to evaluate thelocal calculation ability, such as the CPU processing speed of thecomputer, the memory capacity, etc., and send the evaluated localcalculation ability information to the calculation unit.

The configuration parameter unit is configured to extract aconfiguration parameters from the configuration item module, such as thenumber of threads, the maximum activation time of the threads, etc., andsend the configuration parameter or the operated configuration parameterto the calculation unit.

The calculation unit is configured to calculate task payload accordingto the quantity and/or capacity of the data files evaluated by the dataevaluation unit, the local calculation ability evaluated by thecalculation ability evaluation unit and the configuration parameterprovided by the configuration parameter unit, and send the calculatedtask payload to the task assignment unit.

The task assignment unit is configured to assign all the data files tobe processed under all the leaf directories as a plurality of tasksaccording to the task payload, and send the assigned tasks to the threadpool module. Each sub-task is only responsible for the collection andparse work of a subset of all the collected leaf directories.

The data evaluation and task decomposition can be performed according totwo dimensions of quantity or capacity.

In the network management device and data collection apparatus accordingto the embodiments of the present invention, the data files to beprocessed under all the leaf directories under a non-leaf directory anda task is assigned, the work load of each task is dynamically adjustedaccording to the quantity/capacity of the data files to be processed,the computer configuration (i.e. local calculation ability) andconfiguration item parameter, which solves the problem that the workload of each thread is uneven, also solves the problem that the workload of the computer is over large or it is not fully utilized, and alsosolves the problem that the work ranges of multiple processes under thesame non-leaf directory are overlapped to form competition; moreover,the decomposed task merely operates the leaf directories, avoidingfrequent access of files and entering or exiting a plurality of levelsof directories, solving the problem that the efficiency of the magneticIO operation is low. The mode used by data processing is the thread poolplus task queue mode and is managed by the configuration item.

All the processing threads can be monitored uniformly, which solves theproblem that all the processing threads cannot be managed uniformly; andthe number of parsing threads and the maximum activation time thereofcan be adjusted according to different scenarios, reducing theincreasing possibility of abnormality occurrence due to the long timerunning of the thread, and achieving the maximum processing efficiencyunder the premise of ensuring the accuracy rate.

Although some processing time will be lost due to a large amount of fileoperations during task scheduling, the loss brought thereby is far lessthan the time saved by the subsequent processing.

FIG. 4 is a flowchart of a data collection method according to theembodiments of the present invention, and as shown in FIG. 4, the datacollection method in this embodiment includes the following steps.

S1, A network management device collects data files to be processedreported by a network element device.

After the trigger triggers the task scheduling module to start working,the task scheduling module collects leaf directories to be processedunder each non-leaf directory, and the data files to be processedreported by various network element devices are stored under these leafdirectories.

S12, the data files to be processed are evaluated.

The data files under the leaf directory are evaluated, for example, thequantity and/or capacity of the data files can be evaluated.

S13, the local calculation ability is evaluated.

The local calculation ability mainly includes: CPU processing speed ofthe computer, memory capacity, etc.

S14, a configuration parameter is obtained.

The configuration parameter mainly includes: the number of threads, themaximum activation time of the threads, etc.

Steps S12 to S14 have no order limit.

S15, the task payload is calculated.

The task payload of each thread is calculated on the basis of theparameters obtained from steps S12 to S14, and the calculation manner ofthe task payload is as follows.

1) The total data CPU processing time is calculated.

The total data CPU processing time is in particular equal to the totalquantity or total capacity of the data files to be processed divided bythe CPU processing speed.

2) The processing batch is calculated.

The quotient of the total data CPU processing time divided by theproduct of the number of threads and the maximum activation time, andthe quotient of the total capacity of the data files to be processeddivided by the memory capacity are firstly calculated, and then thebigger one from the two quotient is taken as the processing batch, andthe expression is as follows:

processing batch=Max (the total data CPU processing time/(the number ofthreads*the maximum activation time), the total capacity of thedata/memory capacity), wherein, Max represents to take the maximum.

3) The task payload is calculated.

The task payload of each thread is the total data capacity divided bythe processing batch and then divided by the number of threads, and theexpression is as follows:

the load of each thread=the total data capacity/the processing batch/thenumber of threads.

Of course, the above rule may not be suitable for every scenario, andother calculation methods are also suitable.

S16, the data files to be processed are assigned as a plurality of tasksbased on the task payload, and each assigned task is added into the taskqueue.

In this case, the contents of the task are merely parsing the data filesunder the leaf directories covered by this task.

S17, the tasks are extracted from the task queue one by one andprocessed.

After having received an activation notification, the thread poolextracts tasks from the task queue one by one and processes the same,and the task processing result can be sent to the subsequent processingmodule.

During the task processing, the processing on the data file can beparsing the data file, and the execution result is data to be writteninto the database or data to be forwarded.

S18, subsequent operations are performed.

The threads of the subsequent processing module continuously extractdata from the data queue and carries out subsequent operations; if thereis no data in the data queue, the thread will try again afterhibernating for some time.

The particular embodiments involved in steps S11 to S18 in this methodare not only limited to scenarios such as network administratorperformance, alerting, inverse construction, but all the scenarios whichneed to collect data can use them. Hereinafter, the data collectionembodiments under three scenarios of network administrator performance,alerting, and inverse construction will be introduced respectively.

Embodiment I

As to collection of performance data by network administrator: thenetwork element device reports the performance data files to the networkmanagement device, and the data collection apparatus in the networkmanagement device collects the performance data and writes the same intothe database.

Various modules are constructed in the data collection apparatus in thisembodiment, which is in particular as follows.

A. Construct a Collection Trigger Module.

The collection trigger module can be achieved by a timer and is used forawaking the task scheduling module to start working.

B. Construct a Configuration Item Module.

In particular, an XML configuration file and the parse class thereof aredefined, and the characteristics in this embodiment relate to thefollowing relevant parameters: the number of threads in the thread poolmodule, the maximum thread activation time and the data evaluationdecomposition granularity (quantity or capacity), and provide theparameter collection function.

In the actual application, all the parameters are written into the XMLconfiguration file, and these parameters needs to be added into thesoftware by way of the parse class.

C. Construct the Task Scheduling Module.

1) the task scheduling module uses the number of threads of theconfiguration item and the maximum activation time;

2) a data evaluation unit is constructed and configured to evaluate thetotal capacity of the data files;

3) a calculation ability evaluation unit is constructed and configuredto evaluate the CPU processing speed and memory capacity of thecomputer;

4) a calculation unit is constructed and configured to calculate thetask payload of the process according to the corresponding calculationrule and assign all the collected data files to each task according tothe task payload of the thread; and the running result is the assignedtask (i.e. implemented task interface), and the assigned tasks are addedinto the task queue in the thread pool.

D. Define and Implement the Task Interface

The function of processing the performance data file under the networkelement is implemented in the task interface.

The task interface can further add the result after having parsed thedata file into the data queue of the subsequent processing module, andthe processing result is the subsequent processing data. The subsequentprocessing data actually refers to the result after the task parses thedata file, and it may need further procedure so as to be called assubsequent processing data.

E. Construct a Thread Pool Module

The thread pool has to use the “the number of threads of the thread poolmodule”, the recommendation value of which is 20, and “the maximumthread activation time”, the recommendation value of which is 10 min, ofthe configuration item.

The thread pool module can add the assigned tasks into the task queue;then extract tasks from the task queue one by one, process the same, andoutput the processed tasks.

Preferably, if some tasks need subsequent processing, such as operationof being written into the database, then the thread pool module outputsthe processed task to the subsequent processing module.

F. Construct the Subsequent Processing Module

Define a data queue and construct a data cache mechanism; and the dataqueue is used for storing data which need subsequent processing.

Construct a processing process, which process is an ever running processand extracts data from the data queue for processing; if it is anoperation of writing into the database, the execution of this processcan store the data into a database relevant table by way of the storageprocess of the database.

Embodiment II

As to the processing of the network management device constructing datainversely: the network management device initiates an inverseconstruction operation toward a plurality of network element devices,the network element device reports the inverse construction data filesto the network management device, and the data collection module in thenetwork management device collects the inverse construction data andwrites the same into the database.

In this case, inverse construction refers to reporting the configurationdata on the network element device to the network management device inthe manner of FTP (file transfer protocol), and the network managementdevice collects these configuration data and finally writes the sameinto the database. The inverse construction operation includes: thenetwork management device issues a command to the network elementdevice, and the network element device reports the configuration data tothe network management device, and the network management devicecollects the configuration data and writes the same into the database.

The basic method of this embodiment is similar to that of embodiment I,and the difference between it and that of embodiment I lies in:

a. the configuration item module employs quantity evaluation and taskdecomposition, where, the reason is that it is not interesting how bigthe capacity is during inverse construction, but is only interesting howmany configuration data need to be collected;

b. the data evaluation unit in the task scheduling module returns thetotal quantity of the data files;

c. the trigger condition in the trigger module is the inverseconstruction operation instead of timer.

Embodiment III

As to the collection of the network management alarm synchronizationdata: the network management device initiates an alarm synchronizationoperation toward a plurality of network element devices, the networkelement device reports the alarm synchronization data to the networkmanagement device, and the data collection module in the networkmanagement device collects the alarm synchronization data and writes thesame into the database.

The basic method of this embodiment is similar to that of embodiment I,and the difference between it and that of embodiment I lies in:

a. the configuration item module employs quantity evaluation and taskdecomposition;

b. the data evaluation unit in the task scheduling module returns thetotal quantity of the data;

c. the trigger condition in the trigger module is the alertingsynchronization operation instead of timer.

The method in this embodiment is suitable for all the “manager-personbeing managed” model environments, and as long as there are data to bereported to the manager by the person being managed, this method can beused.

Above description is only to illustrate the preferred embodiments butnot to limit the present invention. Various alternations and changes tothe present invention are apparent to those skilled in the art. Thescope defined in claims shall comprise any medication, equivalentsubstitution and improvement within the spirit and principle of thepresent invention.

INDUSTRIAL APPLICABILITY

As compared to the related art, the present invention eliminates thepossibility of resource competition, improves the task processingefficiency, improves the IO operation efficiency of entering or exitingthe directory for the magnetic disk, improves the data collectionfunction efficiency without losing system efficiency, reduces thepossibility of abnormality appearing during the long time running of thethread, and avoids the situation that the work load of the computer isover large or it is not fully utilized.

1. A data collection method, comprising: a network management devicecollecting data files to be processed reported by a network elementdevice; assigning the data files to be processed as a plurality oftasks; and adding the plurality of tasks into a task queue andextracting tasks from the task queue one by one for processing.
 2. Themethod as claimed in claim 1, wherein the step of assigning the datafiles to be processed as a plurality of tasks comprises: evaluatingquantity or capacity of the data files to be processed; evaluating localcalculation ability; obtaining local configuration parameters;calculating task payload according to the quantity or capacity of thedata files to be processed, the local calculation ability and theconfiguration parameter; and assigning the data files to be processed asthe plurality of tasks based on the task payload.
 3. The method asclaimed in claim 2, wherein the local calculation ability comprises:local CPU processing speed and memory capacity.
 4. The method as claimedin claim 2, wherein the configuration parameter comprises: a number ofthreads and maximum activation time of the threads.
 5. A networkmanagement device, comprising: a data collection apparatus, which isconfigured to: obtain data files to be processed reported by a networkelement device; assign the data files to be processed into a pluralityof tasks; add the plurality of tasks into a task queue, and extracttasks from the task queue one by one for processing.
 6. The networkmanagement device as claimed in claim 5, wherein the data collectionapparatus is configured to assign the data files to be processed intothe plurality of tasks by way of the following manner: evaluatingquantity or capacity of the data files to be processed; evaluating localcalculation ability; obtaining local configuration parameters;calculating task payload according to the quantity or capacity of thedata files to be processed, the local calculation ability and theconfiguration parameters; and assigning the data files to be processedas the plurality of tasks based on the task payload.
 7. The networkmanagement device as claimed in claim 6, wherein the local calculationability comprises: local CPU processing speed and memory capacity. 8.The network management device as claimed in claim 6, wherein theconfiguration parameters comprises: a number of threads and maximumactivation time of the threads.
 9. A data collection apparatus,comprising: a task scheduling module being configured to obtain datafiles to be processed reported by a network element device, assign thedata files to be processed into a plurality of tasks, and send theplurality of tasks to a thread pool module; and the thread pool modulebeing configured to add the plurality of tasks received into a taskqueue, and extract tasks from the task queue one by one for processing.10. The data collection apparatus as claimed in claim 9, wherein thetask scheduling module comprises: a data evaluation unit beingconfigured to collect data files to be processed reported by a networkelement device, evaluate quantity or capacity of the data files to beprocessed, and send the evaluated quantity or capacity to a calculationunit; a calculation ability evaluation unit being configured to evaluatelocal calculation ability and send the evaluated local calculationability information to the calculation unit; a configuration parameterunit being configured to obtain configuration parameters and send theobtained configuration parameters to the calculation unit; thecalculation unit being configured to calculate task payload according tothe quantity or capacity of the data files to be processed, the localcalculation ability and the configuration parameters, and send thecalculated task payload to a task assignment unit; and the taskassignment unit being configured to, after receiving the task payload,assign the data files to be processed as the plurality of tasks based onthe task payload, and send the plurality of tasks to the thread poolmodule.
 11. The data collection apparatus as claimed in claim 10,wherein the local calculation ability comprises: local CPU processingspeed and memory capacity.
 12. The data collection apparatus as claimedin claim 10, further comprising a configuration item module beingconfigured to define a configuration parameter, wherein theconfiguration parameter comprises: a number of threads and maximumactivation time of the threads; the configuration parameter unit beingconfigured to obtain the configuration parameters from the configurationitem module.
 13. The method as claimed in claim 3, wherein theconfiguration parameter comprises: a number of threads and a maximumthread activation time.
 14. The network management device as claimed inclaim 7, wherein the configuration parameter comprises: a number ofthreads and a maximum thread activation time.
 15. The data collectionapparatus as claimed in claim 11, further comprising a configurationitem module being configured to define a configuration parameter,wherein the configuration parameter comprises: a number of threads and amaximum thread activation time; the configuration parameter unit beingconfigured to obtain the configuration parameter from the configurationitem module.